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No studies of Sheltered Instruction Observation Protocol® (SIOP®) that fall within the scope of the 
English Language Learners review protocol meet What Works Clearinghouse (WWC) evidence stan- 
dards. The lack of studies meeting WWC evidence standards means that, at this time, the WWC is 
unable to draw any conclusions based on research about the effectiveness or ineffectiveness of SIOP® 
on English language learners. Additional research is needed to determine the effectiveness or 
ineffectiveness of this intervention. 

Program Description 1 

The Sheltered Instruction Observation Protocol ® (SIOP®) is a framework for planning and delivering instruction in 
content areas such as science, history, and mathematics to English language learners as well as other students. 

The goal of SIOP® is to help teachers integrate academic language development into their lessons, allowing stu- 
dents to learn and practice English as it is used in the context of school, including the vocabulary used in textbooks 
and lectures in each academic discipline. Using this planning framework, teachers modify the way they teach so 
that the language they use to explain concepts and information is comprehensible to these students. SIOP® is 
intended to be applicable at levels of education from pre-K through community college. The SIOP® model con- 
sists of instructional strategies that cover eight aspects of lesson design and delivery: lesson preparation, building 
background, comprehensible input, strategies, interaction, practice and application, lesson delivery, and review and 
assessment. The instructional strategies address the academic and linguistics needs of English language learn- 
ers. In most cases, teachers receive professional development on the SIOP® model before using it to modify their 
lessons. This review of SIOP® focuses on research that examines its impact on the learning of English language 
learners in grades K-8. 

Research 2 

The WWC identified 32 studies of SIOP® for English language learners that were published or released between 
1983 and 2012. 

Seven studies are within the scope of the English Language Learners review protocol but do not meet WWC 
evidence standards. 

• Four studies did not establish that the comparison group was comparable to the intervention group prior to the 
start of the intervention. Three of these used a quasi-experimental design, and the other was a randomized con- 
trolled trial with high attrition. 

• Three studies include only one unit— for example, school — in one condition, which makes it impossible to attri- 
bute the observed effect solely to SIOP®. 

Twenty-two studies are out of the scope of the English Language Learners review protocol because they have an 
ineligible study design. 
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• Eighteen studies do not use a comparison group design, a regression discontinuity design, or a 
single-case design. 

• Four studies are literature reviews or meta-analyses. 

Three studies are out of the scope of the English Language Learners review protocol for reasons other than 
study design. 

• Two studies do not use a sample aligned with the protocol— the sample includes less than 50% English 
language learners or does not fall within the acceptable grade range. 

• One study does not implement the intervention in a way that falls within the scope of the review— the 
intervention is bundled with other components. 
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Endnotes 

1 The descriptive information for this program was obtained from a publicly available source: the program’s website (http://www.cal. 
org/siop, downloaded November 2012). The WWC requests developers to review the program description sections for accuracy from 
their perspective. The program description was provided to the developer in November 2012, and we incorporated feedback from the 
developer. Further verification of the accuracy of the descriptive information for this program is beyond the scope of this review. The 
literature search reflects documents publicly available by June 2012. 

2 This report has been updated to include reviews of 25 studies that have been released since 2009. (The previous report was released 
in June 2009.) Of the additional studies, 20 were not within the scope of the protocol, and five were within the scope of the protocol 
but did not meet evidence standards. The current disposition for Gammon (2004) differs from the previous intervention report because 
version 2.1 of the English Language Learners protocol now accepts studies with samples of at least 50% English language learners, 
whereas in the past, the cutoff was 60%. However, although the sample is appropriate, the study remains ineligible due to issues with 
the design as described in the current disposition. A complete list and disposition of all studies reviewed are provided in the refer- 
ences. The studies in this report were reviewed using the evidence standards from the WWC Procedures and Standards Handbook 
(version 2.1), along with those described in the English Language Learners review protocol (version 2.2). The evidence presented in 
this report is based on available research. Findings and conclusions may change as new research becomes available. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 

Single-case design 
Standard deviation 


Statistical significance 


Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent of 
evidence levels are given in the WWC Procedures and Standards Handbook (version 2.1). 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the research 
design and the magnitude, statistical significance, and consistency in findings. The criteria for the 
ratings of effectiveness are given in the WWC Procedures and Standards Handbook (version 2.1). 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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